I've one question:
Which of the following situations means that the problem is with retaining the additional parameters?

• The parameters were added to the subnet node and they disappear when creating an HDA from the subnet node.
  Note, Houdini by default add these additional parameters to the HDA as shown in this demo.

  2024-11-15.18-34-39.mp4

• The additional parameters disappears when loading the HDA or creating it from tab menu.

  2024-11-15.18-41-45.mp4
  2024-11-15.18-47-03.mp4

I'll assume it's the second one. I thought we already have an issue for it other than the linked issue #99 but I was wrong.

I checked the issue. so the parameters will disappear when using the parameter interface to add them.
However, using the type properties to add those parameters, keeps them on the node.

Maybe the parameter interfaces is something related to the node instance itself not the node the type.
and the type properties changes the actual type.

So, enabling Save Spare Parameters also saves the extra parameters added by the AYON HDA creator plugin.

Maybe, I can destroy AYON extra parameters before extraction and re-add them at the end of extraction? (while enabling the save spare parameters of course).

Maybe the parameter interfaces is something related to the node instance itself not the node the type.
and the type properties changes the actual type.

Yes - that is correct.

I tested by publishing one version with custom ordered menu and the other with the menu and some float attributes, It seems that the parameter is still not updated as expected. (It could be that I am using 19.5 for test)

I tested by publishing one version with custom ordered menu and the other with the menu and some float attributes, It seems that the parameter is still not updated as expected. (It could be that I am using 19.5 for test)

updated_parameter_not_shown_in_Houdini.mp4

This issue is shown in my demo (the second video in the PR description).

I tested by publishing one version with custom ordered menu and the other with the menu and some float attributes, It seems that the parameter is still not updated as expected. (It could be that I am using 19.5 for test)
updated_parameter_not_shown_in_Houdini.mp4

This issue is shown in my demo (the second video in the PR description).

I also did some retest on that, finding it actually loads the updated parameters for even elder version. I think that makes sense if houdini doesn't really have callback to update parameters by version(show or hide the parameters).

So hear me out.

The only reason why we're imprinting these nodes with these attributes is so that we know what source representation they are, etc. Nothing requires us to store it on the nodes as attributes - we just happen to be doing so because other implementations in Houdini did so too. And hence we could rely on the pre-existing "get_containers" logic that just looked for those attributes.

However, nothing is keeping us from adding an additional loop there that does something like (psuedocode):

for hda in hda_definitions:
    container = get_hda_container_data(hda)
    if container:
        yield container

Or if we do want to start from the node get the representation data from the HDA, but from the node - etc.
Anyway, the question then becomes: why couldn't we store these attributes directly in the HDA definition on publish (maybe not even as attributes)

So hear me out.

The only reason why we're imprinting these nodes with these attributes is so that we know what source representation they are, etc. Nothing requires us to store it on the nodes as attributes - we just happen to be doing so because other implementations in Houdini did so too. And hence we could rely on the pre-existing "get_containers" logic that just looked for those attributes.

However, nothing is keeping us from adding an additional loop there that does something like (psuedocode):

for hda in hda_definitions:
    container = get_hda_container_data(hda)
    if container:
        yield container

Or if we do want to start from the node get the representation data from the HDA, but from the node - etc. Anyway, the question then becomes: why couldn't we store these attributes directly in the HDA definition on publish (maybe not even as attributes)

I tried to manually export the hda node with ayon instance data and, it seems that it gets the parameter data right when I imported.. I tried to use the python to export hda, the parameter data is loaded correctly with the contents locked before exporting. I am not sure if how the parameters are working with the publisher, but I personally think that we can store the attributes into the Extra folder and see although it might not sound good solution.
Speaking of the HDA definition for publish, I do see other issues when we add another folder group other than just Extra folder template, and the extractor only gets the parameters from Folder group published. But this can be a future talk.

Let me add a quick summary.

First of all this is the standard Houdini behavior, it is not relevant to our HDA plugins (to create, publish and load HDA product).
However, using our HDA plugins makes these issues occurs more often.

So, there are couple of things:

Parameters UI and spare parameters

When the HDA is loaded, The Extra parameters folder might change the parameters UI.
Actually, it depends on how the parameters were arranged the first place.
e.g. I got these two results (and they are not so ugly)

No folder for spare parameters	use folder for spare parameters

Update spare parameters on definition change

This one is what really bothers me

Changing the HDA asset definition either from the definition dropdown menu or from the loader. doesn't update the spare parameters accordingly.
It most likely happen because HDAs by default doesn't have a callback to update the spare parameters.
BUT, they will show up correctly if the node is removed and re-created.

The best way to avoid this issue is by adding the parameters using the type properties.

Also, keep in mind, that when changing the definition, the Extra folder is not updated accordingly. mentioned here #12

Mixing parameters (type properties) and spare parameters

This one can result in weird lookin parameters UI especially when keep changing the asset definition.
This also is avoided by deleting and re-creating the node.

Here's how it looks like when removing and re-creating.

Also, keep in mind, that when changing the definition, the Extra folder is not updated accordingly. mentioned here #12

This is only the case because:

1. The attributes are not part of the HDA definition
2. The loader on update updates the representation id value - which in turn becomes a parameter override (so even if it were part of HDA definition it would still now have an override).

If the attributes were part of the HDA definition (Type Properties of the HDA) then we'd never need to change the attributes at all from the loader - we would just need to switch the HDA version itself.

However, I wanted to clarify - what I meant with my earlier post was: "Can't we avoid attributes altogether?" - Like, can't we as part of the HDA defintion itself (not in Type Properties, but in metadata elsewhere) store this representation id, etc. on publish (so it's a static part of the HDA definition itself). That way, loading the HDA - in what form whatsoever, even if loaded e.g. manually without AYON it would still have the imprinted containerisation and could be detected as such?

If that's completely impossible, I also want to stress that by no means is it necessary that these attributes are in a folder, and/or are visible. As such, again on publish of the HDA definition, we can add to its type properties (as we "save the definition" and publish it) the relevant metadata we need as hidden parameters (or even one 'blob data' parameter if we prefer that) where we store what we need and later look back for to find the loaded containers.

I'm completely confused by this issue described here @MustafaJafar - the parameters should not change, at all. What's up with that 'juggling of parameters'? The HDA definition should just have a set of attributes (Type Attributes) and that's it. And on load/update we preferably do as LITTLE as possible so we can rely as much as possible on Houdini. So, at best, all we need to 'detect' the representation is completely defined in the HDA definition - and not something we containerize on load.

As far as I know, to add the Extra parameters to the HDA definition, I can think of two solutions:

• we will need to know the value of these parameters (e.g representation id) before saving the file. (and we will add it and delete it afterwards)
• maybe after the publishing, we can open the published file and add these parameters/set their values.

Honestly, I can't think of a clean way to do it.

Also, for reference, I'm removing the Extra parameters before saving the file and re-adding them after saving the HDA file.
Maybe at this point I can add more parameters or convert spare parameters to be parameters.

I'm completely confused by this issue described here @MustafaJafar - the parameters should not change, at all. What's up with that 'juggling of parameters'? The HDA definition should just have a set of attributes (Type Attributes) and that's it. And on load/update we preferably do as LITTLE as possible so we can rely as much as possible on Houdini. So, at best, all we need to 'detect' the representation is completely defined in the HDA definition - and not something we containerize on load.

This is an Edge case, but once someone hits it, it'll be very annoying.

This happens automatically by Houdini. If my HDA has parameters and spare parameters.
and the parameters differs per different version, Houdini will do its default behavior where it moves the spare parameters to a folder called spare and put the parameters in a folder called standard.

to trigger this effect, you can simply publish HDA with different parameters and use manage to change the version or you can just change it from the asset definition toolbar.

This happens automatically by Houdini. If my HDA has parameters and spare parameters.
and the parameters differs per different version, Houdini will do its default behavior where it moves the spare parameters to a folder called spare and put the parameters in a folder called standard.

So technically, this has nothing to do with us - but is an issue(?) on the Houdini side itself. Or is there reason this is happening to us specifically?

So technically, this has nothing to do with us - but is an issue(?) on the Houdini side itself. Or is there reason this is happening to us specifically?

Because we are making it easier to publish HDAs with spare parameters supported, this behavior will show up more often.

So technically, this has nothing to do with us - but is an issue(?) on the Houdini side itself. Or is there reason this is happening to us specifically?

Because we are making it easier to publish HDAs with spare parameters supported, this behavior will show up more often.

Also for reference, [From Develop Branch] I can publish HDA with parameters (that were add using type properties) and the parameters will exist when loading the HDA!

Also, I've found a nasty bug when I accidently named my parameter to one of the names used by imprint.

During load error happened on Product: "random_SopHDA" Representation: "hda" Version: 2

Error message: The attempted operation failed.
Reserved parameter 'name' already exists

Traceback (most recent call last):
  File "E:\Ynput\ayon-core\client\ayon_core\tools\loader\models\actions.py", line 740, in _load_representations_by_loader
    load_with_repre_context(
  File "E:\Ynput\ayon-core\client\ayon_core\pipeline\load\utils.py", line 325, in load_with_repre_context
    return loader.load(repre_context, name, namespace, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\Ynput\ayon-houdini\client\ayon_houdini\plugins\load\load_hda.py", line 59, in load
    lib.imprint(hda_node, data)
  File "E:\Ynput\ayon-houdini\client\ayon_houdini\api\lib.py", line 380, in imprint
    node.setParmTemplateGroup(parm_group)
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.370/houdini/python3.11libs\hou.py", line 17114, in setParmTemplateGroup
    return _hou.OpNode_setParmTemplateGroup(self, parm_template_group, rename_conflicting_parms)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
hou.OperationFailed: The attempted operation failed.
Reserved parameter 'name' already exists

This happens automatically by Houdini. If my HDA has parameters and spare parameters.
and the parameters differs per different version, Houdini will do its default behavior where it moves the spare parameters to a folder called spare and put the parameters in a folder called standard.

So technically, this has nothing to do with us - but is an issue(?) on the Houdini side itself. Or is there reason this is happening to us specifically?

Not sure if it is really Houdini side of thing as we can actually export the parameters when using python shell to do some simple hda export. The best solution is to reach out the Sidefx for support :).
I am thinking if the context has changed when publishing. (Not sure if remove and add back Extra folder attributes to the HDA could be considered as the changes)

Tagging @BigRoy @moonyuet @antirotor for visibility.
Hey,
A quick summary:
Houdini doesn't have a callback to update the spare parameters when changing the HDA definition.
This is the standard Houdini behavior. and this issue happens outside of AYON integration.

Before this PR, we didn't enable saving spare parameters toggle. So, the issue was avoided.
But, Currently, this issue will happen more often.

Suggested solutions:

1. Add a callback to update the spare parameters on definition change. (I'm not sure if such an event exists and still don't know how to query the spare parameters). But, this won't change how Houdini will mess up the parameters layout if user has combination of standard and spare parameters.
2. Convert spare parameters to standard parameters secretly before publishing. But, should spare parameters kept spare parameters?
3. Add a validator that alerts the users about spare parameters with action that converts spare parameters to standard parameters. But at this point what will be the point of enabling save spare parameters.
4. We don't change the publishing but change the loading, where we delete and recreate the node, reconnecting inputs and outputs and set the values. but, I'm not sure if this will break something. (as far as I can tell this will skip a lot of hassle)

In a call with Mustafa we went through the PR and changes - we ended up going down this route:

• We are not enforcing the "Save Spare Parameters" on the HDA definition - but with this PR do allow a houdini power user to enable it manually inside the HDA definition and make sure the custom "Extra" publish attributes from AYON itself (from the Creator) will then be force excluded from the publish.
• We added support to "Promote spare parameters" to type properties on initial HDA creation, because that's what houdini does by default and most likely the main source of confusion for most users - maybe even @krishnaavril too? Once it's a HDA definition, the user can still choose to either add type properties or spare parameters, etc. The promotion of the spare parameters on Create can be disabled by a new pre create attribute.

• We fixed some creation logic where potentially if a subnet was the first in your selection (when having multiple nodes selected) it would incorrectly make an HDA of just that subnet instead of collapsing all together.
• We changed/fixed logic that if the user selected one node and it was already an HDA that it would still be collapsed - now it will make just that HDA publishable.

Also the UX issues in Houdini's UI mustafa showed here we will ignore - since it's just houdini behavior outside of our scope and not an issue from what we are doing at all.

@moonyuet could you try again?

It works on my side too.
One minor cosmetic thing.. on updating HDA, should we remove the extra folder and re-append it so it always at the end?